This invention relates to the technical field of a method and a device for extracting principal subjects from film recorded images and other original images which are suitable for use with photoprinters and otherwise.
Heretofore, the images recorded on photographic films such as negatives and reversals (which are hereunder referred to simply as xe2x80x9cfilmsxe2x80x9d) have been commonly printed on light-sensitive materials (photographic paper) by means of direct (analog) exposure in which the film image is projected onto the light-sensitive material to achieve its areal exposure.
A new technology has recently been introduced and this is a printer that relies upon digital exposure. Briefly, the image recorded on a film is read photoelectrically, converted to digital signals and subjected to various image processing operations to produce image data for recording purposes; recording light that has been modulated in accordance with the image data is used to scan and expose a light-sensitive material to record a latent image, which is subsequently developed to produce a (finished) print. The printer operating on this principle has been commercialized as a digital photoprinter.
In the digital photoprinter, images can be processed as digital image data to determine the exposing conditions for printing; therefore, the digital photoprinter is capable of performing effective image processing operations such as the correction of washed-out highlights or flat shadows due to the taking of pictures with backlight or an electronic flash, sharpening and the correction of color or density failures and this enables the production of high-quality prints that reproduce images of the quality that has been impossible to achieve by the conventional direct exposure technique. In addition, not only the assembling of images and the splitting of a single image into plural images but also the compositing of characters can be performed by processing the image data and, as a result, prints can be outputted after various editing and/or processing operations have been performed in accordance with specific uses. Outputting images as prints (photographs) is not the sole capability of the digital photoprinter; the image data can be supplied into a computer or the like and stored in recording media such as a floppy disk; hence, the image data can be put to various non-photographic uses.
Having these features, the digital photoprinter is basically composed of an image input unit and an image output unit. The image input unit has a scanner (image reading apparatus) that reads the image on a film photoelectrically and an image processing apparatus that processes the captured image to produce output image data. The image output unit has a printer (image recording apparatus that records a latent image on a light-sensitive material by scan exposing it in accordance with the image data supplied from the image input unit and a processor (developing apparatus) that performs development and other necessary processing on the exposed light-sensitive material to produce a print.
In the scanner, reading light issuing from a light source is allowed to be incident on a film, from which projected light bearing the image recorded on the film is produced and focused by an imaging lens to form a sharp image on an image sensor such as a CCD sensor; the image is then captured by photoelectric conversion and sent to the image processing apparatus as data for the image on the film (as image data signals) after being optionally subjected to various image processing steps.
In the image processing apparatus, image processing conditions are set on the basis of the image data captured with the scanner and image processing as determined by the thus set conditions is performed on the captured image data and the resulting output image data for image recording (i.e., exposing conditions) are sent to the printer.
In the printer, if it is of a type that relies upon exposure by scanning with an optical beam, the latter is modulated in accordance with the image data sent from the image processing apparatus and deflected in a main scanning direction as the light-sensitive material is transported in an auxiliary scanning direction perpendicular to the main scanning direction, whereby a latent image is formed as the result of exposure (printing) of the light-sensitive material with the image bearing optical beam. In the processor, development and other processing as determined by the light-sensitive material are performed to produce a print (photograph) reproducing the image that was recorded on the film.
If the original image contains a human individual, his or her face is the most important element of processing with photoprinters whether they are digital or of the conventional analog type (the two types are hereinafter collectively referred to as xe2x80x9cphotoprintersxe2x80x9d) and the quality and evaluation of the reproduced image, or print, are greatly affected by how beautifully the face of the human individual is finished.
To deal with this problem, the conventional photoprinters extract the face (face region) of a human individual from the image on a film and determines exposing conditions that enable the face to be finished in the desired way. In the case of a digital photoprinter, such exposing conditions are image processing conditions for producing output image data and in the case of an analog photoprinter, they are exemplified by the amount of exposing light and the amounts by which color filters are to be inserted.
Particularly in the case of a digital photoprinter which is capable of very flexible image processing through the processing of image data, the information on the face is important for setting image-dependent optimal image processing conditions and performing the appropriate face extraction is the way that enables high-quality prints to be produced in a more consistent manner.
Under these circumstances, various methods of subject extraction have so far been proposed for extracting various kinds of principal image subjects. To mention a few examples, skin color extraction using the color hues and saturations of an image and contour extraction by detecting the edge portions of an image are known.
However, the state of the images on films varies with the conditions under which the images were taken, such as whether they were shot with an electronic flash or backlight, or whether the films were overexposed or underexposed or whether color failure occurred. Hence, no single method of extraction that has been proposed to date is capable of ensuring consistent and precise face extraction from various states of images. The precision of face extraction can be improved by combining a plurality of methods of extracting image subjects. However, in order to perform consistent and precise face extraction from various states of images, many methods of extraction have to be combined and the time of performing the necessary processing and calculations to extract the desired principal image subject will inevitably increase to cause undesired effects such as the decrease in the efficiency of print production.
As already mentioned, the state of the images on films varies with the conditions under which the images were taken, such as whether they were shot with an electronic flash or backlight. In addition, the state of the principal subject in a particular image such as its size and orientation also varies with the image, particularly the scene in it.
In ordinary photoprinters, the subject of an image is extracted by many methods searching in four directions through the image and the results of extraction by the respective methods are combined to extract (and evaluate) the principal subject of the image, thereby ensuring that the principal image subject can be extracted with satisfactory precision from many and diverse images.
However, this requires a prolonged time to extract the desired principal image subject, which is one of the reasons for the lower efficiency of print production. A further problem with the use of many methods in extracting the principal image subject is that if several methods fail to extract the correct principal subject, the quality of the final image is adversely affected rather than being improved.
The present invention has been accomplished under these circumstances and has as a first object providing a method for extracting principal image subjects which, when used with digital and analog photoprinters alike, can extract the intended principal image subject with high precision and satisfactory efficiency from various states of images.
Second object of the invention is to provide a device for implementing the above method.
Third object of the invention is to provide a method for extracting principal image subjects which, when used with digital and analog photoprinters alike, can extract the principal image subjects from various states of images and principal subjects with satisfactory efficiency and precision, and further achieve an improvement in the efficiency of print production.
The first object of the invention can be attained by its first aspect which is a method for extracting a principal image subject comprising the steps of:
performing a plurality of subject extractions having different characteristics;
synthesizing subjects extracted by the respective subject extractions; and
extracting the principal image subject from a result of a synthesis if the result of said synthesis satisfies a specified criterion of judgment; but
if the result of said synthesis does not satisfy the criterion of judgment, performing an auxiliary image subject extraction having a different characteristic after said synthesizing and extracting the principal image subject using the result of said synthesis and a result of extraction by said auxiliary image subject extraction.
In a preferred embodiment, said criterion of judgment is at least one of a degree of overlap between regions that occurs when the subjects extracted by the respective subject extractions are synthesized and a weighting score applied when the subjects extracted by the respective subject extractions are synthesized.
If, in case of extracting the principal image subject from the result of said synthesis, an image characteristic quantity of the extracted principal image subject exceeds a predetermined threshold, said auxiliary image subject extraction is performed and the principal image subject is extracted using the result of said synthesis and the result of extraction by said auxiliary image subject extraction.
Further, if, in case of extracting the principal image subject from the result of said synthesis, more than one principal image subject are extracted and if the image characteristic quantity of said more than one principal image subject has differences in excess of a specified level, said auxiliary image subject extraction is performed and the principal image subject is extracted using the result of extraction by said auxiliary image subject extraction.
In another preferred embodiment, said subject extractions are at least two operations selected from among skin color extraction, face contour extraction and circular shape extraction, and said auxiliary image subject extraction is at least one operation selected from among hairs-on-head extraction, face""s internal structure extraction, body extraction, non-background extraction and circular shape extraction.
The second object of the present invention can be attained by its second aspect which is a device for extracting a principal image subject, comprising:
a plurality of extracting units which are cascaded and at least one of which comprises:
a plurality of extracting subunits which are processed in parallel and which perform subject extractions having different characteristics; and
a condition evaluating section which evaluates conditions for said subject extractions based on results in the respective extracting subunits; and wherein
said condition evaluating section performs weighting on respective extracting regions of subjects extracted by said extracting subunits, and/or evaluation of appropriateness of said subjects, as well as selection of at least one of said extracting units which is located downstream and to which extraction processing is applied based on the results thereof.
Preferably, said extracting units comprise at least one unit which has one extracting subunit performing one subject extraction and the condition evaluating section which evaluates the condition for said subject extraction based on the results in the extracting subunit, or at least one unit which has no extracting subunit but only the condition evaluating section which evaluates the conditions for the extracted subjects based on the results in the cascaded precedent extracting unit.
Preferably, said extracting subunits comprises at least one section selected from the group consisting of a skin color extracting section, a face contour extracting section, a circular shape extracting section, a body extracting section, an eye extracting section, a face""s internal structure extracting section, a hairs-on-head extracting section and a non-background extracting section, as well as an extracting section for performing at least two of a skin color extraction, a face contour extraction, a circular shape extraction, a body extraction, an eye extraction, a face""s internal structure extraction, a hairs-on-head extraction and non-background extraction.
Preferably, said condition evaluating section comprises a synthesis/result evaluating section which synthesizes candidate subjects having different characteristics as extracted in the plurality of extracting subunits processed in parallel, and which judges whether the results of the subject extractions are appropriate or not based on the extracting regions of said synthesized candidate subjects.
Preferably, the condition evaluating section in the last one of the cascaded extracting units is a principal image subject evaluating section for extracting the principal image subject.
Moreover, the present invention also provides a method for extracting a principal image subject, comprising:
a plurality of extracting steps which are cascaded, at least one of said-plurality of extracting steps comprising:
a plurality of extracting substeps processed in parallel for performing subject extractions having different characteristics; and
a condition evaluating substep for evaluating conditions for said subject extractions based on results in the respective extracting substeps; and wherein
said condition evaluating substep is performing weighting on respective extracting regions of subjects extracted in said extracting substeps, and/or evaluation of appropriateness of said subjects, as well as selection of at least one of said extracting steps which is located downstream and to which extraction processing is applied based on the results thereof.
Preferably, said plurality of extracting steps contain at least one step which has one extracting substep performing one subject extraction and the condition evaluating substep which evaluates the conditions for said subject extraction based on the results in the extracting substep, or at least one step which has no extracting substep but only the condition evaluating substep which evaluates the conditions for the extracted subjects based on the results in the cascaded precedent extracting step.
Preferably, said extracting substeps comprise at least one selected from the group consisting of skin color extracting, face contour extracting, a circular shape extracting, body extracting, eye extracting, face""s internal structure extracting, hairs-on-head extracting, non-background extracting and extraction for performing at least two thereof.
Preferably, said condition evaluating substep comprises synthesis/result evaluating substep synthesizing candidate subjects having different characteristics as extracted in the plurality of extracting substeps processed in parallel, and judging whether the results of the subject extractions are appropriate or not based on the extracting regions of said synthesized candidate subjects.
Preferably, the condition evaluating substep in the last one of the cascaded extracting steps is a principal image subject evaluating substep for extracting the principal image subject.
The third object of the invention can be attained by its third aspect which is a method of extracting a principal image subject, comprising the steps of:
preliminarily classifying a state of a principal subject in an image into a plurality of types;
setting an appropriate parameter for the extraction of the principal subject for each state of the principal subjects;
evaluating the similarity between two images; and,
if similarity is found to exist between the image from which the principal subject is to be extracted and the image from which the principal subject has already been extracted, extracting the principal subject using a parameter for the extraction of the principal subject that corresponds to the state of the principal subject in the image that has already been subjected to the extraction of the principal subject.
In a preferred embodiment, the state of the principal subject in said image is at least one element of the group consisting of size, orientation, position in the image, color distribution and density distribution.
The third object of the invention can also be attained by its fourth aspect which is a method of extracting a principal image subject, comprising the steps of:
extracting a principal subject from an image;
using luminance data on shooting of the image to determine luminance relationship between a region containing the extracted principal subject and its surrounding region, as well as density relationship between the region containing the extracted principal subject and the surrounding region; and
judging for the match between said luminance and density relationships to see if a result of extracting said principal subject is appropriate or not.
If, in the case where an already processed image has been given a special treatment such as color/density adjustment, correction for the light source, one of various enhancement steps and red eye correction in response to an operator""s command, the current image to be processed resembles said processed image, the former is preferably subjected to a similar or comparable special treatment.